Improve stability of InstrumenterUnloadTest#8967
Conversation
| long initialUnloadCount = classLoadingMXBean.getUnloadedClassCount(); | ||
|
|
||
| // load an isolated class which we know can be unloaded after a full GC | ||
| new IsolatingClassLoader().loadClass("jvmbootstraptest.UnloadingChecker$Canary"); |
There was a problem hiding this comment.
Nice. I was considering something similar inside GCUtils.
Specifically, I was thinking a PhantomReference to an isolated Class, then verify unloading via a ReferenceQueue.
That would at least provide some indication that classes were collection rather than a random Object.
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.023 s) : 0, 1023212
Total [baseline] (8.513 s) : 0, 8512978
Agent [candidate] (1.019 s) : 0, 1019354
Total [candidate] (8.521 s) : 0, 8520948
section iast
Agent [baseline] (1.148 s) : 0, 1148443
Total [baseline] (9.158 s) : 0, 9157811
Agent [candidate] (1.154 s) : 0, 1154271
Total [candidate] (9.169 s) : 0, 9169090
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.151 s) : 0, 1151389
Total [baseline] (9.175 s) : 0, 9175351
Agent [candidate] (1.147 s) : 0, 1146530
Total [candidate] (9.13 s) : 0, 9129712
section iast_TELEMETRY_OFF
Agent [baseline] (1.151 s) : 0, 1150672
Total [baseline] (9.209 s) : 0, 9209463
Agent [candidate] (1.149 s) : 0, 1149405
Total [candidate] (9.219 s) : 0, 9218915
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.706 ms) : 0, 682706
BytebuddyAgent [candidate] (681.89 ms) : 0, 681890
GlobalTracer [baseline] (239.441 ms) : 0, 239441
GlobalTracer [candidate] (239.941 ms) : 0, 239941
AppSec [baseline] (55.465 ms) : 0, 55465
AppSec [candidate] (55.699 ms) : 0, 55699
Debugger [baseline] (6.171 ms) : 0, 6171
Debugger [candidate] (6.17 ms) : 0, 6170
Remote Config [baseline] (745.475 µs) : 0, 745
Remote Config [candidate] (729.214 µs) : 0, 729
Telemetry [baseline] (15.152 ms) : 0, 15152
Telemetry [candidate] (11.413 ms) : 0, 11413
section iast
BytebuddyAgent [baseline] (801.043 ms) : 0, 801043
BytebuddyAgent [candidate] (804.106 ms) : 0, 804106
GlobalTracer [baseline] (230.102 ms) : 0, 230102
GlobalTracer [candidate] (232.133 ms) : 0, 232133
IAST [baseline] (27.083 ms) : 0, 27083
IAST [candidate] (27.157 ms) : 0, 27157
AppSec [baseline] (52.281 ms) : 0, 52281
AppSec [candidate] (52.782 ms) : 0, 52782
Debugger [baseline] (5.951 ms) : 0, 5951
Debugger [candidate] (5.991 ms) : 0, 5991
Remote Config [baseline] (592.207 µs) : 0, 592
Remote Config [candidate] (598.967 µs) : 0, 599
Telemetry [baseline] (7.911 ms) : 0, 7911
Telemetry [candidate] (7.959 ms) : 0, 7959
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.578 ms) : 0, 802578
BytebuddyAgent [candidate] (798.934 ms) : 0, 798934
GlobalTracer [baseline] (230.826 ms) : 0, 230826
GlobalTracer [candidate] (229.944 ms) : 0, 229944
IAST [baseline] (25.475 ms) : 0, 25475
IAST [candidate] (27.082 ms) : 0, 27082
AppSec [baseline] (54.386 ms) : 0, 54386
AppSec [candidate] (52.69 ms) : 0, 52690
Debugger [baseline] (6.002 ms) : 0, 6002
Debugger [candidate] (5.952 ms) : 0, 5952
Remote Config [baseline] (615.106 µs) : 0, 615
Remote Config [candidate] (587.567 µs) : 0, 588
Telemetry [baseline] (7.935 ms) : 0, 7935
Telemetry [candidate] (7.923 ms) : 0, 7923
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (802.283 ms) : 0, 802283
BytebuddyAgent [candidate] (800.217 ms) : 0, 800217
GlobalTracer [baseline] (230.965 ms) : 0, 230965
GlobalTracer [candidate] (231.611 ms) : 0, 231611
IAST [baseline] (27.964 ms) : 0, 27964
IAST [candidate] (28.905 ms) : 0, 28905
AppSec [baseline] (51.346 ms) : 0, 51346
AppSec [candidate] (50.509 ms) : 0, 50509
Debugger [baseline] (5.952 ms) : 0, 5952
Debugger [candidate] (6.046 ms) : 0, 6046
Remote Config [baseline] (592.377 µs) : 0, 592
Remote Config [candidate] (595.036 µs) : 0, 595
Telemetry [baseline] (7.882 ms) : 0, 7882
Telemetry [candidate] (7.979 ms) : 0, 7979
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1022066
Total [baseline] (11.159 s) : 0, 11159034
Agent [candidate] (1.025 s) : 0, 1024887
Total [candidate] (11.115 s) : 0, 11115007
section appsec
Agent [baseline] (1.175 s) : 0, 1174576
Total [baseline] (11.266 s) : 0, 11265665
Agent [candidate] (1.174 s) : 0, 1174296
Total [candidate] (11.182 s) : 0, 11181686
section iast
Agent [baseline] (1.15 s) : 0, 1149705
Total [baseline] (11.307 s) : 0, 11307280
Agent [candidate] (1.15 s) : 0, 1150365
Total [candidate] (11.41 s) : 0, 11410419
section profiling
Agent [baseline] (1.291 s) : 0, 1290929
Total [baseline] (11.564 s) : 0, 11563932
Agent [candidate] (1.27 s) : 0, 1269729
Total [candidate] (11.532 s) : 0, 11531825
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.837 ms) : 0, 682837
BytebuddyAgent [candidate] (684.887 ms) : 0, 684887
GlobalTracer [baseline] (240.479 ms) : 0, 240479
GlobalTracer [candidate] (240.827 ms) : 0, 240827
AppSec [baseline] (56.971 ms) : 0, 56971
AppSec [candidate] (56.545 ms) : 0, 56545
Debugger [baseline] (6.143 ms) : 0, 6143
Debugger [candidate] (6.194 ms) : 0, 6194
Remote Config [baseline] (770.79 µs) : 0, 771
Remote Config [candidate] (732.961 µs) : 0, 733
Telemetry [baseline] (11.374 ms) : 0, 11374
Telemetry [candidate] (12.164 ms) : 0, 12164
section appsec
BytebuddyAgent [baseline] (705.926 ms) : 0, 705926
BytebuddyAgent [candidate] (704.323 ms) : 0, 704323
GlobalTracer [baseline] (233.818 ms) : 0, 233818
GlobalTracer [candidate] (234.738 ms) : 0, 234738
AppSec [baseline] (175.672 ms) : 0, 175672
AppSec [candidate] (175.958 ms) : 0, 175958
Debugger [baseline] (5.919 ms) : 0, 5919
Debugger [candidate] (5.882 ms) : 0, 5882
Remote Config [baseline] (621.328 µs) : 0, 621
Remote Config [candidate] (608.888 µs) : 0, 609
Telemetry [baseline] (7.312 ms) : 0, 7312
Telemetry [candidate] (7.319 ms) : 0, 7319
IAST [baseline] (21.744 ms) : 0, 21744
IAST [candidate] (21.874 ms) : 0, 21874
section iast
BytebuddyAgent [baseline] (801.183 ms) : 0, 801183
BytebuddyAgent [candidate] (801.77 ms) : 0, 801770
GlobalTracer [baseline] (230.351 ms) : 0, 230351
GlobalTracer [candidate] (230.921 ms) : 0, 230921
AppSec [baseline] (52.981 ms) : 0, 52981
AppSec [candidate] (53.503 ms) : 0, 53503
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (5.902 ms) : 0, 5902
Remote Config [baseline] (674.806 µs) : 0, 675
Remote Config [candidate] (584.911 µs) : 0, 585
Telemetry [baseline] (7.929 ms) : 0, 7929
Telemetry [candidate] (7.881 ms) : 0, 7881
IAST [baseline] (27.0 ms) : 0, 27000
IAST [candidate] (26.217 ms) : 0, 26217
section profiling
ProfilingAgent [baseline] (106.289 ms) : 0, 106289
ProfilingAgent [candidate] (104.396 ms) : 0, 104396
BytebuddyAgent [baseline] (689.153 ms) : 0, 689153
BytebuddyAgent [candidate] (677.214 ms) : 0, 677214
GlobalTracer [baseline] (365.882 ms) : 0, 365882
GlobalTracer [candidate] (360.799 ms) : 0, 360799
AppSec [baseline] (62.664 ms) : 0, 62664
AppSec [candidate] (61.567 ms) : 0, 61567
Debugger [baseline] (6.215 ms) : 0, 6215
Debugger [candidate] (6.03 ms) : 0, 6030
Remote Config [baseline] (661.138 µs) : 0, 661
Remote Config [candidate] (654.774 µs) : 0, 655
Telemetry [baseline] (8.319 ms) : 0, 8319
Telemetry [candidate] (8.164 ms) : 0, 8164
Profiling [baseline] (106.314 ms) : 0, 106314
Profiling [candidate] (104.42 ms) : 0, 104420
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 1 performance regressions! Performance is the same for 5 metrics, 17 unstable metrics.
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section baseline
no_agent (1.48 ms) : 1468, 1492
. : milestone, 1480,
appsec (2.405 ms) : 2356, 2454
. : milestone, 2405,
iast (2.193 ms) : 2131, 2254
. : milestone, 2193,
iast_GLOBAL (2.232 ms) : 2171, 2294
. : milestone, 2232,
profiling (2.055 ms) : 2005, 2105
. : milestone, 2055,
tracing (2.018 ms) : 1970, 2065
. : milestone, 2018,
section candidate
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (2.403 ms) : 2355, 2452
. : milestone, 2403,
iast (2.188 ms) : 2127, 2249
. : milestone, 2188,
iast_GLOBAL (2.233 ms) : 2171, 2295
. : milestone, 2233,
profiling (2.046 ms) : 1995, 2097
. : milestone, 2046,
tracing (2.009 ms) : 1961, 2056
. : milestone, 2009,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d69b0ffbda, baseline=1.50.0-SNAPSHOT~089cf8fd47
dateFormat X
axisFormat %s
section baseline
no_agent (15.74 s) : 15740000, 15740000
. : milestone, 15740000,
appsec (14.904 s) : 14904000, 14904000
. : milestone, 14904000,
iast (18.961 s) : 18961000, 18961000
. : milestone, 18961000,
iast_GLOBAL (18.186 s) : 18186000, 18186000
. : milestone, 18186000,
profiling (15.23 s) : 15230000, 15230000
. : milestone, 15230000,
tracing (14.564 s) : 14564000, 14564000
. : milestone, 14564000,
section candidate
no_agent (15.517 s) : 15517000, 15517000
. : milestone, 15517000,
appsec (14.918 s) : 14918000, 14918000
. : milestone, 14918000,
iast (18.383 s) : 18383000, 18383000
. : milestone, 18383000,
iast_GLOBAL (18.126 s) : 18126000, 18126000
. : milestone, 18126000,
profiling (15.259 s) : 15259000, 15259000
. : milestone, 15259000,
tracing (15.014 s) : 15014000, 15014000
. : milestone, 15014000,
|
dougqh
left a comment
There was a problem hiding this comment.
I suggested an approach that would probably work without using JMX.
But as long as the test is more reliable, I'm happy with this as is.
|
|
||
| while (System.nanoTime() - startNanos < waitNanos) { | ||
| try { | ||
| GCUtils.awaitGC(); |
There was a problem hiding this comment.
I would call directly System.gc() here
awaitGC is still fundamentally flawed because of weak reference
What Does This Do
Introduces a canary class which we use to detect when we've done enough to trigger class unloading. If we weren't able to trigger any class unloading then we effectively skip this test, because the results are inconclusive.
Motivation
Hopefully less flakiness
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]